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INTRODUCTION 

On  7-9  October  2002,  more  than  100  NATO,  Partnership  for  Peace,  and  Non-NATO  nationals  from 
22  countries  met  in  Warsaw,  Poland  to  discuss  the  role  of  humans  in  intelligent  and  automated  systems. 
Sponsored  by  the  Human  Factors  and  Medicine  Panel  of  the  North  Atlantic  Treaty  Organization’s  Research 
and  Technology  Organization,  the  symposium  participants  discussed  the  problem,  research  approaches  and 
techniques  for  how  automation  technology  can  take  advantage  of  human  strengths  and  compensate  for  human 
disadvantages  and  how  humans  can  take  advantage  of  automated  systems  strengths  and  compensate  for 
automated  systems  disadvantages. 


THEME  /  OVERVIEW 

Automation  may  increase  efficiency,  but  it  also  raises  doubts  about  adequate  human  control  over  automated 
systems  and  making  sure  that  system  effectiveness  is  not  jeopardized.  This  symposium  focused  on  the 
interaction  of  humans  with  a  growing  array  of  automated  functions  and  automated  and  intelligent  systems. 
During  the  symposium,  participants  discussed  how  to  harmonize  the  interactions  of  humans  with  automated 
and  semi-automated  systems  to  increase  overall  mission  performance.  The  symposium  participants  outlined 
recommendations  for  development  of  human-centered  automation  in  military  environments,  addressing  key 
areas  such  as  providing  levels  of  automation  that  are  appropriate  to  levels  of  risk,  examining  procedures  for 
recovery  from  emergencies,  and  ensuring  human  control  of  automation. 


SYMPOSIUM  PROGRAM 

The  Symposium,  co-chaired  by  Dr.  N.  Gershon  (USA)  and  Dr.  K.  Boff  (USA),  consisted  of  an  Opening 
Session,  three  sessions  on  design  philosophy,  two  sessions  and  a  roundtable  discussion  on  design 
methodology,  two  sessions  on  design  evaluation,  and  a  Capstone  Panel  with  open  discussion.  Six  Keynote 
Addresses  were  interspersed  throughout  the  program.  All  told,  1 8  papers  were  presented,  and  5  poster  papers 
were  available  for  review. 


TECHNICAL  EVALUATION 

This  technical  evaluation  is  divided  into  three  parts.  The  first  part  is  an  overview  of  the  problem  as  presented 
in  the  Opening  Session  (not  otherwise  captured  in  this  volume)  and  by  the  Keynote  Speakers  throughout  the 
Symposium.  The  second  part  reviews  the  content  discussed  in  each  of  the  sessions  -  design  philosophy, 
design  methodology,  and  design  evaluation  -  as  presented  by  the  paper  authors.  The  point  is  not  to  provide  a 
summary  of  each  paper  (these  appear  elsewhere  in  the  volume),  but  to  point  out  the  relevance  of  the  paper  to 
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the  problem  statement.  The  third  and  final  part  of  this  report  evaluates  the  content  presented  and  discussed 
throughout  all  sessions  and  the  Capstone  Panel,  and  concludes  with  recommendations  for  action  that  emerged 
during  the  Symposium. 

I:  Problem  Statement 

Opening  Session 

Col.  W.  Tielemans  (NE),  the  Chairman  of  the  Human  Factors  in  Medicine  Panel  noted  that  it  was  an  honor  for 
the  Panel  to  meet  in  Poland.  In  his  view,  NATO’s  RTO  is  the  focal  point  for  the  research  and  technology 
opportunities  available  to  the  new  NATO  member  states  and  the  overall  Partnership  for  Peace. 

MajGen  C.  Mikrut  (POL),  the  Deputy  Commander  of  the  Polish  Air  Force  and  host  of  the  Symposium, 
stressed  the  importance  of  intelligent  and  automated  systems  for  the  new  NATO  countries,  especially  in  the 
training  process.  We  must  all  adjust  to  change,  Mikrut  noted,  both  in  the  role  of  technology  and  in  the  role  of 
humans  in  intelligent  and  automated  systems.  For  these  countries  (and  for  others),  meeting  the  demands  on  the 
armed  forces  with  downsized  resources  and  numbers  depends  on  effective  cooperation  between  humans  and 
machines,  itself  dependent  on  the  wise  investment  in  research  and  technology  in  intelligent  and  automated 
systems.  He  argued  that  selecting  the  right  person  for  the  right  human-machine  system  is  increasingly 
important. 

Col  P.  Rusecki  (POL),  General  Sanitory  Inspector  for  the  Polish  Army,  stressed  the  need  for  automation 
technology  to  improve  the  performance,  health,  safety,  and  well-being  of  military  personnel.  The  demands 
of  the  2T*‘  century  remind  us  that  we  live  in  an  interdependent  world,  all  facing  acute  global  problems 
together.  These  problems  require  collective  solutions,  integrating  efforts  by  countries  with  differing  levels  of 
technology.  Incorporating  intelligent  and  automated  systems  will  help  us  all  to  meet  new  challenges. 

Dr.  N.  Gershon  (USA)  emphasized  that  this  topic  has  been  of  interest  to  scientists,  artists,  filmmakers  and  the 
like,  and  that  we  scientists  can  learn  from  the  perspectives  of  artists  and  filmmakers  and  the  issues  they 
identify.  Take  the  scene  in  the  film  2001  when  the  computer  “Hal”  declares  his  error  to  be  “human-caused.” 
Since  humans  developed  Hal’s  program  and  the  interfaces,  isn’t  Hal  right  in  a  profound  sense?  That  is, 
in  considering  the  role  of  humans  in  human- automated  system  interactions,  we  must  look  beyond  the  human 
user  to  the  human  program  developer.  Gershon  also  challenged  participants  as  follows:  Why  have  there  been 
so  few  successful  implementations  of  intelligent  agents?  How  can  we  tell  when  a  program  that  controls  a 
process  becomes  an  agent?  Also,  do  agents  in  fact  have  to  be  intelligent? 

According  to  Dr.  Ken  Boff  (USA),  rapid  advances  in  intelligent  system  automation,  networking, 
and  communications  technologies  are  likely  to  change  the  nature  of  conflict  and  also  provide  a  few  surprises. 
Early  approaches  at  incorporating  these  technologies  tended  towards  the  substitution  of  new  capabilities 
performed  by  machines  for  traditional  functions  performed  by  humans,  rather  than  reconceptualizing  the 
nature  of  these  functions  to  gain  maximum  advantage.  In  some  instances,  overall  system  effectiveness  has 
been  reduced  rather  than  advanced.  The  technological  intensity  of  the  long-anticipated  “revolution  in  military 
affairs”  portend  fundamental  and  significant  challenges  to  sustaining  and  enhancing  human  effectiveness  in 
military  operations  precisely  in  those  areas  where  human  decision  making  is  central  to  success.  In  such  areas, 
Boff  argues,  we  must  focus  on  the  role  of  humans  in  intelligent  and  automated  systems  to  gain  maximum 
advantage  of  the  overall  human-machine  system. 
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Keynote  Addresses 

In  Keynote  Address  I,  Dr.  J.  Reising  (USA)  discussed  the  proper  role  of  operators  in  uninhabited  military 
(ground,  airborne,  and  underwater)  vehicles.  Such  vehicles  cover  a  spectrum  of  human-machine  interaction, 
from  tele-operated  to  semi-autonomous  systems.  He  argues  that  the  basic  operating  principles  of  varying 
levels  of  autonomy  and  dynamic  function  allocation  will  enable  the  human  operator  and  the  associate  vehicle 
to  form  a  team  consisting  of  two  (or  more)  crewmembers,  at  least  one  human  and  one  electronic.  To  function 
efficiently  and  effectively,  the  team  members  must  trust  each  other.  Since  it  is  not  clear  how  the  machine 
could  or  could  not  trust  the  human  operator,  the  crucial  aspect  of  teambuilding  is  for  the  operator  to  trust  the 
machine.  He  discussed  guidelines  for  how  to  help  build  a  high  quality,  trusting  relationship.  One  key  aspect  is 
the  ability  of  the  human  to  specify  in  advance  the  levels  of  automated  system  authority  for  various  mission 
phases. 

In  Keynote  Address  II,  Prof.  R.  Amalberti  (FR)  reviewed  how  the  use  of  intelligent  and  automated  systems 
were  expected  to  increase  system  performance,  release  humans  from  repetitive  tasks,  and  reduce  routine 
errors.  While  these  goals  have  been  met,  there  are  concerns  about  the  growing  dependency  of  humans  on 
automated  systems  for  cognition,  particularly  during  operations  in  degraded  modes.  Paradoxically, 
the  technological  complexity  of  the  automated  systems  is  reducing  training  time,  thereby  increasing  this 
cognitive  dependency,  particularly  for  novices.  It  is  important  to  identify  in  advance  whether  automated 
system  use  is  designed  to  train  novices  to  become  experts  in  low-risk  situations  (such  as  in  civil  aviation), 
or  to  enable  experts  to  become  super-experts  in  high  risk  situations  (such  as  in  military  aviation).  The  French 
military  leadership,  Amalberti  argues,  has  not  yet  clearly  defined  its  goals  for  the  use  of  automation. 
Amalberti  also  notes  the  shift  in  military  aviation  human  factors  research  from  individual  to  team 
and  systemic  problem  solving  and  training.  Today,  team  and  systemic  performance  is  fast  becoming  the 
limiting  factor  of  performance.  With  the  expanded  span  of  communication  and  controls  now  available  in 
military  aviation,  there  is  an  increased  need  for  joint  human  factor  training  between  pilots,  planners, 
controllers,  and  mechanics. 

In  Keynote  Address  III,  Mr.  R.  Taylor  (UK)  focused  on  the  key  human  factors  issue  today  -  how  to  ensure  the 
appropriate  levels  of  automation  for  military  decision  functions  given  the  need  to  mitigate  the  associated  risks 
for  human  understanding,  prediction,  and  control  of  system  functioning.  He  addressed  the  ability  of  intelligent 
systems  to  decompose  goals  assigned  by  the  human.  Traditionally,  knowledge-based  behavior  has  been  left 
to  humans,  with  automated  systems  involved  in  skill-  and  rule-based  behavior.  Taylor  argues  that  humans 
and  what  he  terms  ‘cognitive  automated  systems’  should  be  partners  in  all  three  types  of  behavior  in  a 
collaborative  vice  merely  a  coordinated  or  cooperative  system.  This  “complementarity  principle”  moves  the 
issue  from  human-machine  interaction  to  human- computer  collaboration  to  ensure  the  optimum  contribution 
of  both  to  military  system  effectiveness.  One  key  driver  of  effectiveness  will  be  the  ability  of  military  systems 
involving  cognitive  automate  systems  to  adapt  to  the  anticipated  changes  in  envisioned  operations  as  well  as 
to  the  needs  of  unanticipated  future  operations. 

In  Keynote  Address  IV,  Mr.  E.  Tamowski  (FR)  focused  on  the  cockpit  automation  philosophy  of  the 
Airbus  Corporation.  Airbus  has  developed  automated  systems  to  play  on  the  strengths  of  the  human  operator 
(e.g.,  “airmanship”  -  intuition  through  experiential  learning,  and  compensate  for  the  operator’s  weaknesses 
(humans  are  non-repetitive  manipulators,  affected  by  stress,  with  a  propensity  to  unpredictability  and  a 
potential  for  laziness).  Automation  can  be  a  complement  to  the  human,  improving  both  flight  safety  and 
efficiency  when  used  where  it  can  do  better  than  the  human  (e.g.,  tactical  vice  strategic  tasks).  But  this  is 
possible  only  if  the  automation  is  tailored  and  adjusted  for  the  human,  and  if  the  human  adapts  to  automation 
through  judicious  training.  Pilots  can  no  longer  be  “lone  rangers,”  Tamowski  believes,  and  must  be  leaders  of 
a  team  that  includes  other  humans  and  the  automation  systems. 
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In  Keynote  Address  V,  Prof.  R.  Onken  (GE)  discussed  cognitive  cooperation  for  the  sake  of  human-machine 
team  effectiveness.  He  identified  the  need  for  the  artificial  cognitive  systems  to  have  a  great  amount  of 
knowledge  in  common  with  the  human  operator.  Particularly  important  is  the  need  for  explicit  knowledge  of 
the  governing  work  team  objectives,  set  by  the  human,  that  underlay  sensitive  interaction  by  the  machine  as  a 
team  mate. 

In  Keynote  Address  VI,  Dr.  J.  Bradshaw  (USA)  provided  an  overview  of  the  basic  principles  and  pitfalls  of 
adjustable  autonomy  and  human-centered  teamwork  and  a  broad  model  of  human-agent  teamwork  grounded 
in  practice.  His  address,  based  on  a  paper  written  with  several  colleagues’  also  covered  the  interim  results  of  a 
study  on  the  problem  of  work  practice  modeling  and  human-agent  collaboration  in  space  applications. 
Bradshaw  argues  that  agent  technology  can  provide  the  means  for  distributed  agent  components  to  engage  in 
high  level  collaboration.  Further,  for  human-agent  teamwork,  agent  technology  can  help  exploit  synergy, 
adjust  autonomy,  expose  relevant  states,  and  avoid  clumsy  automation  by  overcoming  the  limitations  of  direct 
point-and-click  manipulation  interfaces. 

II:  Content  Presented 

Design  Philosophy 

The  three  design  philosophy  sessions  were  chaired  by  Prof  Dr.  X.  van  Norren  (NE),  Dr.  A.  Eeger  (FR), 
and  Dr.  S.  Kirschenbaum  (USA). 

Dr.  R.  Breton  (CA)  presented  a  paper  co-written  with  Dr.  E.  Bosse  (CA)  that  discusses  the  cognitive  costs  and 
benefits  of  automation  {Paper  #1).  They  note  a  paradox  between  the  need  for  technological  support  to  execute 
command  and  control  tasks  adequately  in  complex  environments,  and  the  contribution  of  technology  to  the 
complexity  of  that  environment.  Automating  sub-parts  of  the  decision-making  task,  they  argue,  can  resolve 
this  paradox.  Since  automated  systems  use  deductive  reasoning  and  humans  use  inductive  reasoning,  quality 
human-automated  system  interactions  -  with  a  balance  between  the  complexity  of  automated  systems  and  the 
transparency  of  those  systems  to  the  human  partners  -  should  result  in  a  high-performing  team.  Central  to 
such  quality  system  interactions  is  understanding  on  the  part  of  the  human  of  how  the  system  is  working, 
coupled  with  access  by  the  human  to  the  sources  of  information  (that  are  processed  by  the  automated 
systems).  Hence,  the  importance  of  training  the  human  to  become  a  better  supervisor  of  the  automated  system 
to  improve  overall  system  performance. 

EtCol  Dr.  A.  Worm  (SWF)  focused  on  the  increased  reliance  on  distributed  and  automated  systems 
in  network-centric  military  operations  and  in  emergency  management  {Paper  #2).  In  both  situations,  highly 
qualified  automated  command  support  is  critical.  Worm  argues  that  such  support  is  best  developed  through 
the  use  of  scenarios  in  advanced  distributed  simulations  involving  experienced  operators  and  decision-makers. 

Dr.  N.  Allen  (CA)  presented  a  paper  co-written  with  Dr.  R.  Kessel  (CA)  that  examined  the  roles  of  the 
humans  and  machines  in  decision  aid  strategies  for  target  detection  in  undersea  warfare  {Paper  #3). 
Conventional  automated  detection  systems  fail  when  the  detector’s  abilities  are  not  matched  to  its 
responsibility.  Automated  decision  aids  can  improve  overall  system  performance  if  there  is  a  working  balance 
between  the  responsibilities  and  abilities  for  all  detectors,  human  and  machine,  in  the  system.  Both  the  system 
designer’s  and  human  operator’s  understanding  of  the  human-machine  roles  are  crucial  for  success. 


‘  The  other  authors  include:  M.  Sierhuis,  A.  Acquisti,  P.  Feltovich,  R.  Hoffman,  R.  Jeffers,  D.  Prescott,  N.  Suri,  A.  Uszok,  and 
R.  van  Hoof,  all  from  the  USA. 
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Dr.  B.  Digney  (CA)  discussed  how  humans  can  teach  learning  machines  in  the  control  of  autonomous  land 
vehicles  {Paper  #4).  Humans  combine  or  adapt  knowledge  they  are  taught  and  generate  new  knowledge 
only  when  needed.  To  be  successful,  automated  systems  must  be  able  to  decompose  learned  models  into 
reusable  chunks,  recombining  such  chunks  whenever  possible  in  future  situations.  Such  hierarchical 
learning  control  systems,  if  “taughf  ’  successfully  by  human  operators  through  hierarchical  learning  methods 
(imitation,  instruction,  graduated  complexity,  protected  exploration,  and  graduated  lessons),  may  be  able  to 
adapt  to  situations  quickly  and  with  high  confidence.  With  uninhabited  military  vehicles,  shared  control 
between  the  on-board  autonomous  control  system  and  remote  operators  may  be  necessary  for  more  efficient 
and  effective  performance  levels.  In  such  systems,  the  on-board  autonomous  control  system  controls  the 
vehicle  for  routine  operation,  with  a  remote  operator  assuming  control  during  exceptional  situations. 

Dipl.-Wirt.  Inform.  O.  Mooshage  (GE)  presented  a  paper  co-written  with  Dipl.-Ing.  H.  Distelmaier  (GE) 
on  human- centered  decision  support  for  naval  anti-air  warfare  where  rule-based  decisions  are  insufficient 
{Paper  #5).  Using  automated  systems  to  improve  the  cognitive  processes  of  human  operators  requires  the 
empirical  observation  of  human  knowledge  acquisition.  They  describe  their  plans  for  such  empirical 
observation  in  a  laboratory  test  bed. 

Dr.  R.  Foster  (USA)  presented  a  paper  co-written  with  Dr.  D.  Fletcher  (USA)  asserting  that  research  and 
technology  challenges  of  future  asynchronous,  continuously  available  systems  sets  a  pacing  goalpost  for  other 
systems  {Paper  #6).  Asynchronous  learning  environments  work  so  well,  they  argue,  because  they  allow  the 
individualization  of  learning  sequence,  content,  style,  difficulty,  and  pace.  They  have  been  applied  effectively 
in  military  training  (witness  the  success  of  the  U.S.  Navy’s  Interactive  Multi-sensory  Analysis  Trainer), 
but  can  also  be  applied  to  a  variety  of  other  situations  if  designers  fully  understand  the  tasks  at  hand, 
their  complexity,  and  they  types  of  errors  that  typically  occur.  This  suggest  a  need  to  determine,  preferably 
through  international  cooperative  research  using  standard  metrics  and  practices,  and  re-adjust  the  balance 
between  resources  allocated  to  training  and  resources  allocated  to  performance  aiding  systems. 

Dr.  Reising  presented  papers  on  pilot  automation  philosophies  in  the  American  and  Russian  Air  Forces’ 
ground  collision  avoidance  systems  written  by  Dr.  W.  Albery  (USA)  and  Col.  M.  Khomenko  (RUS), 
respectively  {Paper  #8).  Both  nations  have  developed  successful  systems  that  represent  very  different  design 
philosophies.  Because  U.S.  pilots  consider  in-flight  monitoring  of  pilot  performance  to  be  intrusive, 
the  ground  control  avoidance  system  measures  aircraft  performance  and  automatically  takes  over  flight 
operations  should  certain  parameters  be  met.  By  contrast,  the  Russian  system  entails  detailed  monitoring  of 
pilot  performance  through  measurable  physiological  parameters  (head  position,  breathing  frequency, 
flight  control  stick  pressure,  rudder  pedal  pressure,  etc.),  and  forces  the  pilot  to  actively  over-ride  a  fail-safe 
system  that  would  otherwise  enable  an  operator  in  a  ground  station  to  take  over.  Both  systems  will  work  and, 
if  fully  implemented,  would  save  lives  and  financial  resources  in  the  cost  of  lost  aircraft  during  training. 
How  such  systems  would  operate  during  combat  remains  an  open  question. 


Design  Methodology 

The  two  design  methodology  sessions  were  chaired  by  Ms.  J.  Marsden  (UK)  and  Dr.  S.  Giray  (TU). 

Dr.  B.  Doering  (GE)  presented  a  paper  co-written  with  Mr.  G.  Doerfel  (GE)  and  Mr.  H.  Distelmaier  (GE) 
examining  the  modeling  of  situation  awareness  for  naval  anti-air  warfare  decision  support  {Paper  #10).  It  will 
be  implemented  in  the  test  facility  described  by  Mooshage  and  Distelmaier  {Paper  #5).  The  approach  uses 
knowledge-based  user  interfaces  to  support  an  operator’s  situation  awareness.  They  intend  to  use  the 
developed  models  as  the  basis  for  developing  a  software  specification  with  the  object-oriented  Unified 
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Modeling  Language  that  would,  in  turn,  faeilitate  the  implementation  of  object-oriented  programming 
languages  in  later  development  of  decision  support  systems. 

Dr.  J.  Y.  Grau  (FR)  presented  a  paper  co-written  with  Mr.  L.  Guichard,  Mr.  F.  Drogoul,  Mrs.  S.  Guibert, 
and  Dr.  G.  Gawinowski  (all  of  FR)  examining  the  need  for  improved  man-machine  interfaces  in  air  traffic 
control  systems  {Paper  #11).  Developing  such  “ultra-safe”  systems  require  a  design  methodology  that 
incorporates  multi-disciplinary  approaches.  Such  is  the  case  for  the  experimental  centre  of  the  European 
organization  for  the  safety  of  air  navigation  (Eurocontrol),  which  uses  its  air  traffic  control  operational 
room  simulation  platforms  in  the  validation  process.  The  process  integrates  psycho-physiological,  cognitive, 
and  sociological  human  factors  domains  in  a  common  study. 

Dr.  M.  Young  (USA)  presented  a  paper  co-written  with  Dr.  R.  Eggleston  (USA)  exploring  work-centered 
decision  support  {Paper  #12).  The  development  of  digital  nervous  systems  is  threatening  to  overwhelm  users 
with  information.  Real-time  enterprises  have  responded  to  this  threat  by  adopting  “digital  dashboards”  - 
displays  that  continually  update  enterprise  information.  Young  and  Eggleston  argue  that  digital  dashboard 
technology  does  not  go  far  enough  in  utilizing  the  capabilities  of  digital  nervous  systems.  Since  users  typically 
work  multiple  work  threads  simultaneously,  computer  interfaces  must  shift  from  the  desktop  metaphor  to 
work-support  at  the  level  of  the  work  threads.  Work-centered  support  uses  intelligent  agents  to  find,  fuse, 
format,  and  present  actionable  information,  generally  using  geographic  information  systems. 

Dr.  E.  Bosse  (CA)  presented  a  paper  co-written  by  Mr.  S.  Paradis  (CA),  Dr.  R.  Breton  (CA),  Dr.  W.  Elm 
(USA),  and  Dr.  S.  Potter  (USA)  exploring  a  cognitive  system  engineering  approach  to  modeling  dynamic 
human  decision-making  activities  in  intelligent  and  automated  systems  for  naval  warfare  {Paper  #13).  In  an 
applied  cognitive  work  analysis  approach,  the  focus  shifts  from  what  is  to  be  displayed  to  how  information 
should  be  displayed  given  an  understanding  of  human  presentation  techniques  and  attributes.  This  approach 
has  an  added  benefit  of  reducing  analytic  costs  and  improving  analysis-design  efficiency. 

Dr.  M.  Cook  (UK)  presented  a  paper  co-written  by  Ms.  C.  Angus,  Ms.  C.  Adams,  and  Mr.  C.  Cranmer  (all  of 
the  UK)  examining  strategies  for  using  cognitive  automated  system  to  enhance  electronic  warfare  situational 
awareness  {Paper  #14).  Automation  helps  mediate  the  world  for  the  user,  but  at  least  for  electronic  warfare, 
the  weak  element  of  the  partnership  is  often  the  system  and  not  the  human.  As  users  increasingly  see  the 
world  only  as  mediated  by  automation  systems,  they  are  less  accommodating  of  unexpected  or  unusual  states 
in  the  automation.  The  key  is  to  maximize  automated  system  universality  (it  does  what  it  says  it  does,  every 
time,  with  no  exceptions)  and  meet  the  human’s  expectations  for  transparency  (keeping  the  operator  informed 
at  the  level  desired  by  the  operator),  criticality  (the  degree  of  risk  involved  in  the  execution  of  a  particular 
action),  and  pace  (supporting  the  human  so  that  the  operation  of  events  are  not  overwhelming). 

Design  Evaluation 

The  two  design  evaluation  sessions  were  chaired  by  Dr.  M.  Cook  (UK)  (standing  in  for  Prof  Paul  Noja  (IT), 
who  was  ill)  and  Dr.-Ing.  B.  Goering  (GE). 

Mr.  S.  Galster  (USA)  presented  a  paper  co-authored  with  Mr.  R.  Bolia  (USA)  and  Dr.  R.  Parasuraman  (USA) 
discussing  the  need  to  examine  the  effects  of  unreliable  automation  on  system  performance  {Paper  #15). 
Such  unreliability  results  from  three  causes:  problems  in  the  use  of  automated  systems  by  human  operators, 
unanticipated  events,  or  interactions  with  other  sub-systems.  They  present  a  qualitative  model  of  human 
interaction  with  automation  to  reduce  the  costs  and  time  of  full  system  integration  testing.  In  this  model, 
automation  tasks  are  assigned  to  each  of  four  stages  of  a  simplified  human  information-processing  model: 
acquisition,  analysis,  decision,  and  action  implementation.  Their  research  suggests  that  unreliable  automation 
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can  have  more  profound  negative  effects  with  greater  automation  use  across  the  stages.  By  contrast, 
even  under  conditions  of  automation  unreliability,  use  of  automation  in  the  acquisition  and  analysis  stages 
alone  improves  performance.  This  suggests  that  over-reliance  on  automation  can  lead  to  automation-induced 
complacency  on  the  part  of  the  operator,  a  risk  that  is  of  more  concern  the  greater  the  degree  of  automation 
unreliability. 

Dr.  B.  Hilburn  (NE)  presented  a  paper  evaluating  human  interaction  with  advanced  air  traffic  automation 
systems  (Paper  #17).  He  reports  on  research  exploring  adaptive  automation  concepts  in  real-time  human-in- 
the-loop  simulations  evaluated  by  objective  psycho-physiological  measures  of  human  performance. 
The  research  shows  workload  benefits  associated  with  adaptive  automation.  However,  subjective  measures 
suggested  a  bias  against  advanced  forms  of  automation.  This  suggests  the  importance  of  operator 
familiarization  with  the  automation  during  phased  training  to  increase  operator  acceptance  of  new  automation. 

Dr.  G.  Wilson  (USA)  presented  a  paper  examining  the  use  of  psycho-physiological  measures  to  classify 
operator  function  states  and  evaluate  when  to  use  adaptive  automated  decision  aids  (Paper  #18). 
When  psycho-physiological  signals  indicate  that  the  operator  is  in  an  unfavorable  state  (i.e.,  states  of  fatigue 
and  inattention,  or  cognitive  overload),  adaptive  aiding  systems  can  be  implemented  in  an  attempt  to  enhance 
overall  performance.  Operator  acceptance  of  this  technology  will  be  based  upon  demonstrated  validity  and 
reliability  to  the  individual  operator. 

Dr.  S.  Kirschenbaum  (USA)  presented  a  paper  arguing  that  automated  systems  must  collaborate  with  the  user 
to  improve  the  ability  for  both  to  understand  the  situation  (Paper  #19).  Without  effective  communication 
between  system  and  user,  user  lack  of  confidence  in  the  system  will  lead  to  errors.  Since  experts  evaluate  the 
reliability  of  automated  data  by  checking  it  against  “raw”  data,  systems  that  incorporate  the  representation 
match  hypothesis  -  representations  of  uncertainty  should  match  the  representations  required  for  problem 
solving  -  can  help  make  human-automated  system  communications  more  effective.  The  more  cognitive  work 
an  operator  has  to  go  through  to  match  internal  to  external  representation  (or  vice  versa),  the  more  difficult  the 
task  will  be. 

D.  H.  Veltman  (NE)  presented  a  paper  co-authored  with  D.  A.  Oving  (NE)  discussing  the  use  of  three- 
dimensional  map  displays  for  improving  operator  control  of  unmanned  aerial  vehicles  (Paper  #21). 
Their  research  demonstrates  that  operator  performance  in  the  control  of  uninhabited  aerial  vehicles,  as 
evaluated  with  subjective  and  psycho-physiological  measures,  improves  with  three  dimensional  map  displays, 
especially  when  the  quality  of  camera  images  was  decreased  by  time  delays  or  low  update  rates.  Operators 
perceive  workload  to  be  reduced,  though  this  perception  was  not  confirmed  by  the  psycho-physiological 
workload  measures. 

Dipl.-Inform.  B.  Trouvain  (GE)  presented  a  paper  co-written  with  Dipl.-Ing.  H.  Wolf  (GE)  on  the  use 
of  multiple,  semi-autonomous  mobile  robots  in  military  domains  for  reconnaissance,  mine  detection, 
and  operations  in  hazardous  environments  (Paper  #22).  Different  levels  of  robot  autonomy  will  affect 
operator  performance;  along  with  the  overall  number  of  robots  involved  -  the  greater  the  number  of  robots, 
the  greater  the  required  degree  of  autonomy  for  operator  effectiveness.  Designing  effective  multi-robot  control 
interfaces  is  an  especially  challenging  human  factors  design  task.  Without  sophisticated  automated  support, 
supervising  multi-robot  systems  larger  than  two  robots  is  hard  to  realize  if  tight  monitoring  is  required. 

Ill:  Thematic  Evaluation 

In  the  Opening  Session,  Boff  suggested  that  human  factor  designers  must  help  to  reconceptualize  the  nature  of 
functions  to  be  allocated  to  human  and  machine  to  gain  maximum  advantage  for  the  overall  system.  Reising 
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{Keynote  IT)  goes  further  to  suggest  that  the  nature  of  the  human-maehine  relationship  must  be 
reeoneeptualized  from  that  of  user  and  information  /  eognitive  provider  to  that  of  fully  engaged,  mutually 
supportive  members  of  a  elose-knit  team.  We  ean  infer  from  this  that  the  foeus  should  be  on  the  role  of 
teaming  in  human- automated  system  interaetions,  with  the  role  of  humans  as  part  of  that  team  -  the  prineipal 
human  faetors  eoneern  -  a  derivative  question. 

One  of  the  proposed  key  innovations  over  the  past  deeades  in  human-human  teaming  has  been  the 
introduction  of  computer  and  communication  innovations  to  enable  virtual,  synchronous  and  asynchronous 
collaboration  to  improve  efficiency  and  effectiveness.  But  while  the  potential  for  such  innovations  is  great, 
success  has  been  rare  -  seven  out  of  ten  deployments  of  such  “groupware”  has  failed,  with  failure  often 
attributed  to  incompatibility  of  collaboration  with  existing  business  practices,  culture,  sociology,  and  the  way 
humans  tend  to  collaborate.  Research  into  groupware  deployments  suggests  that  success  is  ninety  percent 
people  interactions  with  each  other  and  with  processes,  and  only  ten  percent  the  technology  involved.  Can  we 
infer  similar  difficulties  with  human-automated  system  interaction  and  teaming? 

According  to  Taylor  {Keynote  III),  the  role  of  humans  in  intelligent  and  automated  systems  can  be  evaluated 
by  looking  at  capability  (what),  cognition  (why),  autonomy  (who  and  how),  and  context  (where  and  when). 
In  this  section,  we  examine  this  question  by  reviewing  the  arguments  presented  and  discussed  in  the 
Symposium  by  these  elements;  references  to  views  of  Symposium  participants  without  the  connection  to 
specific  Keynote  addresses  or  papers  derive  from  the  Capstone  Panel  discussion. 

Capability:  What  Roles  for  each  Team  Member? 

According  to  Reising  {Keynote  1),  the  team  arrangement  design  philosophy  has  rarely,  if  ever,  been  carried 
out.  Ideally,  a  division  of  roles  between  the  two  (or  more)  team  members  occurs  by  the  designer  taking 
account  of  the  strengths  and  weakness  of  each  and  assigning  roles  accordingly  (i.e.,  “Fitts’  list”  after  Paul 
Fitts).  Reising  argues  that  “fittsian”  functional  allocation  is  largely  a  myth  and  rarely  applied  in  system  design 
and  development.  Rather  than  an  integral  part  of  an  automated  system,  he  suggests  that  the  operator  has 
become  more  of  an  observer,  taking  control  of  the  system  when  emergencies  occur. 

Taylor  {Keynote  III)  believes  that  such  supervisory  roles  are  appropriate  for  relatively  stable,  static  and 
predictable  “closed-loop”  systems.  But  in  the  military  battle-space  characterized  by  change,  uncertainty, 
and  complex  environments,  the  human  role  is  to  provide  cognition  of  the  system  in  the  sense  of  thinking, 
conceiving  and  reasoning.  The  human  must  ensure  that  subordinate  team  members  -  in  this  instance  the 
machine(s)  -  are  fully  conversant  with  the  command  intent  of  the  human  operator. 

In  short,  as  Taylor  argues,  the  challenge  in  human-machine  teaming  is  to  provide  information  and  decision 
systems  that  protect  and  preserve  the  human  operator’s  key  role,  and  that  augments  and  enhances  the  user’s 
cognition  rather  than  replacing  the  user  in  complex  decision-making.  To  preserve  a  human-centric  system, 
a  sensible  architecture  for  distributed  multi-agent  intelligent  systems  must  involve  adjustable  levels  of 
machine  autonomy  based  on  a  strong  socio-technical  basis.  As  detailed  by  Albery  and  Khomenko  {Paper  #8), 
accounting  for  socio-technical  bases  can  lead  to  very  different  architectures  with  comparative  levels  of 
success. 

Cognition:  Why  Intelligent  and  Automated  Systems  should  be  Part  of  the  Team? 

Bradshaw  {Keynote  VI)  argues  that  a  better  approach  than  functional  allocation  is  to  determine  how  tasks  can 
best  be  shared  by  humans  and  agents  working  in  concert,  what  he  terms  an  “un-Fitts”  list.  In  such  an 
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approach,  the  boundary  between  human  and  agent  initiative  can  be  dynamically  adjusted,  with  tasks 
reassigned  as  necessary.  Taylor  {Keynote  III)  refers  to  this  as  “the  complementarity  principle,”  where  both 
human  and  automation  contribute  to  skill-based,  rule-based,  and  knowledge  based  behavior.  The  ultimate 
situation  would  be  when  the  boundary  between  agents  and  people  disappears  altogether.  To  Bradshaw,  the  key 
concept  involves  shared  knowledge,  goals,  and  intentions  that  function  as  the  glue  that  binds  team  members 
together.  He  emphasizes  the  need  to  incorporate  theory  from  social  sciences  when  considering  human-agent 
interaction,  and  to  observe  interactions  empirically  to  build  more  effective  models. 

Onken  {Keynote  V)  argues  that  cognitive  human-machine  cooperation  as  opposed  to  mere  interaction  has  great 
similarities  to  human  work  team  cooperation.  Cognitive  cooperation  occurs  when  members  of  a  team  are  not 
bound  to  a  firm  allocation  of  tasks.  Rather,  given  access  to  all  information  available,  members  are  capable  of 
adopting  the  roles  of  other  cooperating  units  working  on  shared  objectives.  For  human-machine  cognitive 
cooperation,  success  is  highly  depending  on  the  capability  of  the  artificial  cognitive  units  to  found  its  activities 
on  a  valid  model  of  the  human  operator. 

Taylor  {Keynote  III)  suggests  that  ultimately,  intelligent  and  automated  systems  must  be  part  of  the  team  to 
ensure  that  humans  have  the  technical  tools  and  means  for  human  problem-solving  in  both  envisioned 
operations  and  unanticipated  future  operations. 

Autonomy:  Who  Should  Have  Control,  and  How? 

Is  the  human  supervising  the  machine,  or  the  machine  the  human?  How  do  we  prevent  the  human  from 
becoming  the  associate  of  the  intelligent  agent?  The  key  question,  Boff  suggests,  is  whether  the  operator  is  so 
dependant  upon  the  information  processed  by  the  agent  that  the  information  provided  will  constrain  the 
decision  option  space.  This  certainly  happens  in  human  teaming  situations,  where  the  information  available  to 
the  team  members,  or  provided  by  one  team  member  to  another,  constrains  the  outcomes.  Or,  maybe  it  is  also 
a  question  of  trust  (see  later  in  this  section). 

Reising  believes  that  the  contract  specified  by  the  human  is  the  key  thing.  In  this  fashion,  the  operator  knows 
when  and  where  he  or  she  will  be  in  charge.  Taylor  argues  that  the  human  operator  must  be  near  the  basic 
sources  of  information.  Kirschenbaum  suggests  that  this  is  a  critical  characteristic  of  experts,  who  are  aware 
of  the  fact  that  they  depend  on  information  fed  to  them  by  others  who  may  be  less  expert.  It  is  the  job  of 
human  and  social  sciences  researchers  to  press  upon  intelligent  and  automation  system  designers  the  need 
for  experts  to  get  their  “feet  wet”  in  the  raw  data  to  facilitate  their  ability  to  get  a  sense  of  the  information, 
and  whether  the  options  being  fed  are  being  constrained  incorrectly.  Gershon  cautions  that  once  the  human 
operator  gets  used  to  the  information  as  processed  by  the  agent,  it  will  be  difficult  for  that  operator  to  process 
the  raw  input  (by  analogy,  going  back  to  pre- calculator  days,  even  to  do  long  division).  And  Bradshaw  warns 
that  once  the  introduction  of  automated  systems  is  successful,  the  workload  might  increase  such  that  the 
human  is  no  better  off. 

Others  see  solutions  to  this  conundrum  in  the  overall  capacity  of  humans  in  the  broader  system.  Ober  suggests 
that  that  the  entire  chain  of  human-machine  interaction  extends  beyond  the  occasions  when  the  human  sits  in 
front  of  the  console.  Both  he  and  Onken  suggest  that  the  overall  work  system  must  include  the  human  above 
the  human  interfacing  with  the  automated  systems.  Hilbum  argues  that  by  integrating  system  designers  into 
this  broader  human  system  early,  the  human(s)  above  and  the  human  operator(s)  can  build  trust  in  these 
cognitive  keyholes.  This  will  enable  Reising’s  contractual  approach,  involving  the  human  above  the  operator 
as  well,  to  be  successful.  Taylor  suggests  that  the  blurring  or  merging  of  the  designer-operator  role  is 
important.  The  solution,  he  believes,  rests  in  real-time  human  engineering  (including  design)  of  the  system 
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control,  configuration  processes  as  well  as  system  mission  function.  With  an  evolving,  adaptive  organic 
system,  the  issue  becomes  system  configuration  management,  command  and  control,  and  in-use  system 
verification,  validation,  and  utility  assurance  -  all  part  of  the  continuing  human  command  function. 

Context:  Where  and  When  should  Intelligent  and  Automated  Systems  have  Autonomy? 

Reising  {Keynote  I)  focuses  on  the  development  of  trust  relationships  for  successful  cognitive  human-machine 
cooperation.  He  argues  that  operator  trust  in  the  machine  team  member  must  be  built  up  gradually  in  stages, 
moving  from  the  predictability  of  the  machine’s  behavior  (first  stage),  through  trust  based  on  dependability 
accumulated  through  behavioral  evidence  (second  stage),  to  faith  that  the  team  members  have  in  each  other’s 
dependability  in  the  future  (third  stage).  To  do  so,  the  human  operator  must  be  engaged  in  designing  the 
machine’s  prime  directives,  specifying  its  level  of  autonomy,  and  selecting  the  machine’s  displays  to  conform 
to  the  operator’s  mental  model  of  the  system.  Only  through  such  a  process  can  a  knife-edge  balance 
be  maintained  between  providing  sufficient  data  exchange  without  swamping  the  human  team  member, 
and  achieving  sufficiently  autonomy  for  the  machine  without  alienating  the  human  manager.  Otherwise, 
as  Digney  alludes  {Paper  #4),  the  machines  will  not  be  accepted  (that  is,  will  not  “make  the  team”). 

Automated  systems,  Amalberti  {Keynote  IT)  argues,  can  only  get  “better”  if  operators  train  with  them  as  part 
of  a  team.  Digney  suggests  that  it  is  easier  to  trust  automated  systems  when  we  think  of  them  as  capable  of 
learning.  Certainly,  automated  systems  can  only  be  useful  if  they  are  used,^  and  operator  use  depends  on  trust. 
As  discussed  at  the  Symposium,  trust  is  earned  in  stages,  and  mistrust  is  learned.  Unreliable  automation  can 
significantly  affect  trust,  just  as  unreliable  humans  can  affect  perceptions  of  automation.  Hilburn’s  research 
with  air  traffic  management  automation  {Paper  #1 7)  supports  this  argument.  Prof.  E.  Hollnagel  suggests 
that  without  awareness  of  how  the  automated  system  works,  humans  will  not  use  such  systems  effectively. 
And  Wilson  {Paper  #18)  stresses  the  importance  of  individual  operator  experience  with  the  automated 
assistant  prior  to  operational  use. 

According  to  Tamowski  {Keynote  IV),  this  is  inherent  in  the  role  of  the  human  as  team-leader,  to  coordinate 
and  organize  all  tasks  to  be  executed  by  each  team  member,  human  or  automated  system.  Being  a  team-leader, 
the  human  has  to  adapt  to  his  team  members,  requiring  that  he  understand  them.  Otherwise,  the  human  will 
take  control  and  do  it  himself,  thereby  risking  overall  system  efficiency  and  effectiveness  and,  in  extreme 
cases,  safety.  As  an  example,  he  described  a  failure  in  trust  situation,  partially  due  to  a  failure  to  account 
adequately  for  socio-technical  issues  (the  “shut  up  gringo”  comment  by  a  Spanish-speaking  pilot  to  the 
English-speaking  automated  ground  avoidance  warning  system  seconds  before  a  catastrophic  incident). 

Inferring  from  Tamowski,  training  programs  should  be  defined  to  help  the  human  become  an  effective  team- 
leader.  Otherwise,  as  suggested  by  Breton  and  Bosse  {Paper  #1),  the  human  as  adequate  system  supervisor 
(a  more  passive  role  than  leader)  may  prevent  the  human  from  building  an  appropriate  mental  model  of  the 
situation  essential  for  the  recovery  of  system  failures. 

In  a  human-machine  team,  some  argue  that  the  machine  should  take  on  a  persona  to  aid  in  the  social 
interaction  with  the  operator.  While  Young  and  Eggleston  {Paper  #12)  want  the  make  the  agents  more 
cognitive,  they  argue  that  the  introduction  of  social  agents  could  reintroduce  mental  shifts  into  work  as  the 
user  shifts  from  focusing  on  the  task  to  “socializing”  with  the  agent.  They  suggest  that  a  more  important 
concern  is  the  ability  of  the  human  team  member/leader  to  observe  of  the  cognitive  elements  of  the  agent  team 
member.  Paradis  et  al.  {Paper  #13)  report  that  Canada  has  explicitly  designed  transparency  into  their 


^  As  discussed  in  RTO-MP-057,  Usability  of  Information  in  Battle  Management  Operations,  RTO-MP-057,  the  RTO  Human 
Factors  and  Medicine  Panel  Symposium  held  in  Oslo,  Norway,  10-13,  April  2000,  available  at  http://www.rta.nato.int/. 
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automation  decision  support  systems  to  improve  the  trust  aspect  for  the  human  team  member/leader. 
The  issues  of  agent  persona  and  observability  are  both  empirical  concerns  that  part  of  the  human  factors 
research  agenda. 

Hilburn’s  research  {Paper  #17)  stresses  the  importance  of  choosing  “correct”  test  subjects  -  i.e.,  operators 
willing  to  explore  the  utility  of  automated  systems  and  consider  different  operational  approaches  given  the 
presence  of  automation.  Inferring  from  by  Albery  and  Khomenko  {Paper  #8),  using  American  pilots  in  a  test 
of  Russian  ground  collision  avoidance  systems  may  not  be  an  appropriate  test  of  such  systems.  Finally, 
as  discussed  in  the  Capstone  Panel  and  in  the  poster  board  sessions,  it  may  be  possible  to  measure  trust 
disposition  through  human  factors  research  into  operator  functional  states  of  psycho-physiological  condition. 

In  sum,  human-machine  teaming  is  important  at  all  stages  in  the  development  and  implementation  of 
intelligent  and  automated  systems.  However,  it  is  during  the  developmental  phases  that  varying  levels  of 
automated  system  autonomy  can  best  be  explored. 


CONCLUSIONS  AND  RECOMMENDATIONS 

During  the  Symposium,  it  became  clear  that  there  is  no  consensus  (at  least  among  attendees)  on  the  role  of 
humans  in  intelligent  and  automated  systems.  Bradshaw  {in  Keynote  VI)  did  present  a  set  of  desiderata  for 
human-centered  systems  that  could  form  the  basis  for  such  a  consensus  (see  the  box,  below^),  at  least  in  the 
view  of  the  author  of  this  Technical  Evaluation  Report. 


Premise:  Humans  are  responsible  for  outcomes  in  human-machine  systems. 

Axiom:  Humans  must  be  in  command  of  human-machine  systems. 

Corollaries:  Humans  must  be  actively  involved  in  the  processes  undertaken  by  these  systems..., 
adequately  informed  of  human-machine  system  processes...,  [and]  able  to  monitor  the  machine 
components  of  the  machine  components  of  the  system.  The  activities  of  the  machines  must 
therefore  be  predictable.  The  machines  must  also  be  able  to  monitor  the  performance  of  the 
humans.  Each  intelligent  agent  in  a  human-machine  system  [people  and  software  agents]  must 
have  knowledge  of  the  intent  of  the  other  agents. 


Based  on  the  Symposium,  there  should  be  two  additional  corollaries.  First,  as  particularly  emphasized  by 
Allen  and  Kessel  {Paper  #3),  there  should  be  a  balance  of  responsibilities  and  abilities  for  all  elements  - 
human  and  machine  -  of  the  system.  The  strategy  to  develop  this  balance  must  be  clear  to  system  developers, 
operators,  and  technical  planners. 

Second,  as  Gershon  stressed  in  the  Opening  Session,  the  role  of  humans  as  designers  of  automation  systems, 
not  just  as  users  or  mission  experts,  is  of  critical  importance.  Worm  {Paper  #2)  emphasizes  the  need  for  close 
cooperation  between  practitioners,  policymakers,  researchers,  developers,  and  engineers.  Foster  and  Fletcher 
{Paper  #6)  emphasize  the  need  for  the  NATO  /  Partnership  for  Peace  community  to  share  information  through 
a  commonly  accessible  database  and  to  develop  guidelines  and  standards  for  technology-based  education, 
training,  and  performance  aiding.  The  human  factors  community  has  the  opportunity  and  an  obligation  to  lead 
its  members  into  the  future  promised  by  these  technological  applications. 


^  The  citation  to  C.E.  Billings’  work  is  in  Bradshaw’s  Keynote  VI  (as  reference  #8)  elsewhere  in  this  volume. 
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